The present invention relates generally to the art of sensors and displays. It finds particular application in vision systems for operators of manned and unmanned vehicles and is illustrated and described herein primarily with reference thereto. However, it will be appreciated that the present invention is also amenable to surveillance and other tele-observation or tele-presence applications and all manner of other panoramic or wide-angle video photography applications.
Although it has been possible to collect panoramic images and even spherical images for a number of years, it has not been possible to simultaneously acquire and display data panoramically, at its true resolution, in real-time, as three-dimensional (3-D) stereoscopic images. Nor has it been possible to share non-coincident stereo views of the outside of a vehicle. The lack of these capabilities has severely hampered the ability to implement adequate operator interfaces in those vehicles that do not allow the operator to have direct view of the outside world, such as fighting vehicles like tanks and armored personnel carriers, among many other applications. Personnel often prefer to have themselves partially out of the vehicle hatches in order to gain the best visibility possible, putting them at risk of casualty. In the case of tanks, the risk to such personnel includes being hit by shrapnel, being shot by snipers, getting pinned by the vehicle when it rolls, as well as injuring others and property due to poor visibility around the vehicle as it moves.
Previous attempts at mitigating these problems include the provision of windows, periscopes, various combinations of displays and cameras, but none of these has provided a capability that mitigates the lack of view for the operators. Hence, operators still prefer direct viewing, with its inherent dangers. Windows must be small and narrow since they will not withstand ballistics and hence provide only a narrow field of view. Windows also let light out, which at night pinpoints areas for enemy fire. Periscopes have a narrow field of view and expose the operator to injury, e.g., by being struck by the periscope when the vehicle tosses around. Periscopes may also induce nausea when operators look through them for more than very short periods. Previous attempts with external cameras and internal displays similarly induce nausea, provide a narrow or limited field of view, do not easily accommodate collaboration among multiple occupants, endure significant lag times between image capture and display thereby causing disorientation for the users, do not provide adequate depth perception, and, in general, do not replicate the feeling of directly viewing the scenes in question. Further, when a sensor is disabled, the area covered by that sensor is no longer visible to the operator. Hence as of 2005, vehicle operators are still being killed and injured in large numbers.
In addition, display systems for remotely operated unmanned surface, sub-surface, and air vehicles suffer from similar deficiencies, thereby limiting the utility, survivability, and lethality of these systems.
The current state of the art involves the use of various types of camera systems to develop a complete view of what is around the sensor. For example, the Ladybug camera from PT Grey, the Dodeca camera from Immersive Media Corporation, and the SVS-2500 from iMove, Inc., all do this with varying degrees of success. These and other companies have also developed camera systems where the individual sensors are separated from each other by distances of many feet and the resulting data from the dispersed cameras is again “stitched” together to form a spherical or semi spherical view of what is around the vehicle. Most of these cameras have accompanying software that allows a user to “stitch” together the images from a number of image sensors that make up the spherical camera, into a seamless spherical image that is updated from 5 to 30 times per second. Accompanying software also allows one to “de-warp” portions of the spherical image for users to view in a “flat” view, without the distortion caused by the use of very wide-angle lenses on the cameras that make up the spherical sensors. These systems are generally non-real-time and require a post-processing step to make the images appear as a spherical image, although progress is being made in making this process work in real-time. Unfortunately, tele-observation situations such as viewing what is going on outside of a tank as it is being operated require a maximum of a few hundred milliseconds of latency from image capture to display. Present systems do not provide a stereo 3-D view and, hence, cannot replicate the stereoscopic depth that humans use in making decisions and perceiving their surroundings.
Furthermore, the fielded current state of the art still generally involves the use of pan-tilt type camera systems. These pan-tilt camera systems do not allow for multiple users to access different views around the sensor and all users must share the view that the “master” who is controlling the device is pointing the sensor towards.
Accordingly, the present invention contemplates a new and improved vision system and method wherein a complete picture of the scene outside a vehicle or similar enclosure is presented to any number of operators in real-time stereo 3-D, and which overcome the above-referenced problems and others.